Large%2Dscale robust visual codebook construction.

Melog.
We calculate the capacity of a multiple-antenna wireless link with M-transmit and N-receive antennas in a Rician fading channel. We consider the standard Rician fading channel where the channel coefficients are modeled as independent circular Gaussian random variables with non-zero means (non-zero specular component). The channel coefficients of this model are constant over a block of T symbol periods but, independent over different blocks. For such a model, the capacity and capacity achieving signals are dependent on the specular component. We obtain asymptotic expressions for capacity in the low and high SNR (ρ) scenarios. We establish that for low SNR the optimum signal is determined completely by the specular component of the channel and beamforming is the optimum signaling strategy. For high SNR, we show that Rayleigh optimal signaling (the signal that ignores the specular component) achieves the optimal rate of increase of capacity with respect to log ρ. Further, as ρ →∞, the signals that are optimum for Rayleigh fading channel are also optimum for Rician channel in the case of coherent commumincations and in the case of non-coherent communications when T →∞. Finally, we show that the the number of degrees of freedom of a Rician fading channel is atleast as much as that of a Rayleigh fading channel namely K(1 − K T) where K = min{M,N,⌊T/2⌋}.
ACM workshop on mobile cloud media computing.

Image retagging.

Real%2Dtime large scale near%2Dduplicate web video retrieval.


Tagging tags.
Abstract. While the Semantic Web requires a large amount of structured knowledge (triples) to allow machine reasoning, the acquisition of this knowledge still represents an open issue. Indeed, expressing expert knowledge in a given formalism is a tedious process. Less structured annotations such as tagging have, however, proved immensely popular, whilst existing unstructured or semi-structured collaborative knowledge bases such as Wikipedia have proven to be useful and scalable. Both processes are often regulated through social mechanisms such as wiki-like operations, recommendations, ratings, and collaborative games. To promote collaborative tagging as a means to acquire unstructured as well as structured knowledge we introduce the notion of Extreme Tagging, which describes systems which allow the tagging of resources, as well as of tags themselves and their relations. We provide a formal description of extreme tagging followed by examples and highlight the necessity of regulatory processes which can be applied to it. We also present a prototype implementation.

Contextual image retrieval model.

Optimizing kd%2Dtrees for scalable visual descriptor indexing.

Content%2Daware Ranking for visual search.

Compact projection Simple and efficient near neighbor search with practical memory requirements.

Learning to combine multi%2Dresolution spatially%2Dweighted co%2Doccurrence matrices for image representation.

Social Image Search with Diverse Relevance Ranking.

Dynamic Video Collage.

Image search by concept map.

Retagging social images based on visual and semantic consistency.

Interactive image search by 2D semantic map.

Active Reranking for Web Image Search.

Visual quality assessment for web videos.

AdOn toward contextual overlay in%2Dvideo advertising.

GameSense game%2Dlike in%2Dimage advertising.

Metric learning with feature decomposition for image categorization.

Visual query suggestion Towards capturing user intent in internet image search.

Image Search Result Summarization with Informative Priors.

NLVS a near%2Dlossless video summarization system.

Learning semantic distance from community%2Dtagged media collection.

Accessible image search.
Images on the Web present a major accessibility issue for the visually impaired, mainly because the majority of them do not have proper captions. This paper addresses the problem of attaching proper explanatory text descriptions to arbitrary images on the Web. To this end, we introduce Phetch, an enjoyable computer game that collects explanatory descriptions of images. People play the game because it is fun, and as a side effect of game play we collect valuable information. Given any image from the World Wide Web, Phetch can output a correct annotation for it. The collected data can be applied towards significantly improving Web accessibility. In addition to improving accessibility, Phetch is an example of a new class of games that provide entertainment in exchange for human processing power. In essence, we solve a typical computer vision problem with HCI tools alone.
Near%2Dlossless video summarization.

Tag refinement by regularized LDA.

Smart batch tagging of photo albums.

Ranking model adaptation for domain%2Dspecific search.

Ensemble manifold regularization.

MSRA%2DMM 2.0 A Large%2DScale Web Multimedia Dataset.

Boost search relevance for tag%2Dbased social image retrieval.

Tag quality improvement for social images.

Efficient image and video re%2Dcoloring for colorblindness.

Summarizing tagged image collections by cross%2Dmedia representativeness voting.

Robust Distance Metric Learning with Auxiliary Knowledge.
Most of the existing metric learning methods are accomplished by exploiting pairwise constraints over the labeled data and frequently suffer from the insufficiency of training examples. To learn a robust distance metric from few labeled examples, prior knowledge from unlabeled examples as well as the metrics previously derived from auxiliary data sets can be useful. In this paper, we propose to leverage such auxiliary knowledge to assist distance metric learning, which is formulated following the regularized loss minimization principle. Two algorithms are derived on the basis of manifold regularization and log-determinant divergence regularization technique, respectively, which can simultaneously exploit label information (i.e., the pairwise constraints over labeled data), unlabeled examples, and the metrics derived from auxiliary data sets. The proposed methods directly manipulate the auxiliary metrics and require no raw examples from the auxiliary data sets, which make them efficient and flexible. We conduct extensive evaluations to compare our approaches with a number of competing approaches on face recognition task. The experimental results show that our approaches can derive reliable distance metrics from limited training examples and thus are superior in terms of accuracy and labeling efforts. 1
Robust Distance Metric Learning with Auxiliary Knowledge.
Most of the existing metric learning methods are accomplished by exploiting pairwise constraints over the labeled data and frequently suffer from the insufficiency of training examples. To learn a robust distance metric from few labeled examples, prior knowledge from unlabeled examples as well as the metrics previously derived from auxiliary data sets can be useful. In this paper, we propose to leverage such auxiliary knowledge to assist distance metric learning, which is formulated following the regularized loss minimization principle. Two algorithms are derived on the basis of manifold regularization and log-determinant divergence regularization technique, respectively, which can simultaneously exploit label information (i.e., the pairwise constraints over labeled data), unlabeled examples, and the metrics derived from auxiliary data sets. The proposed methods directly manipulate the auxiliary metrics and require no raw examples from the auxiliary data sets, which make them efficient and flexible. We conduct extensive evaluations to compare our approaches with a number of competing approaches on face recognition task. The experimental results show that our approaches can derive reliable distance metrics from limited training examples and thus are superior in terms of accuracy and labeling efforts. 1
Graph%2DBased Pairwise Learning to Rank for Video Search.

Multiple%2DInstance Active Learning for Image Categorization.

CrowdReranking exploring multiple search engines for visual search reranking.

AdOn an intelligent overlay video advertising system.

Concept representation based video indexing.

Query sampling for ranking learning in web search.

Accommodating colorblind users in image search.

Gamesense.

Tag ranking.
Social annotations on a Web document are highly generalized description of topics contained in that page. Their tagged frequency indicates the user attentions with various degrees. This makes annotations a good resource for summarizing multiple topics in a Web page. In this paper, we present a tag-oriented Web document summarization approach by using both document content and the tags annotated on that document. To improve summarization performance, a new tag ranking algorithm named EigenTag is proposed in this paper to reduce noise in tags. Meanwhile, association mining technique is employed to expand tag set to tackle the sparsity problem. Experimental results show our tag-oriented summarization has a significant improvement over those not using tags.
Learning to tag.
This paper describes RoboTag, an advanced  prototype for a machine learningbased  multilingual information extraction  system. First, we describe a general  client/server architecture used in learning  from observation. Then we give a detailed  description of our novel decision-tree tagging  approach. RoboTag performance for  the proper noun tagging task in English  and Japanese is compared against humantagged  keys and to the best hand-coded  pattern performance (as reported in the  MUC and MET evaluation results). Re-  lated work and future directions are pre-  sented.
Video Content Structure.

Introduction to computer vision and image understanding the special issue on video analysis.

Semi%2Dsupervised kernel density estimation for video annotation.
Abstract—Canonical correlation analysis (CCA) is a powerful tool for analyzing multi-dimensional paired data. However, CCA tends to perform poorly when the number of paired samples is limited, which is often the case in practice. To cope with this problem, we propose a semi-supervised variant of CCA named “semiCCA”that allows us to incorporate additional unpaired samples for mitigating over ttng. The proposed method smoothly bridges the eigenvalue problems of CCA and principal component analysis (PCA), and thus its solution can be computed ef ciently just by solving a single eigenvalue problem as the original CCA. Index Terms—Canonical correlation analysis, semi-supervised learning, generalized eigenproblem, automatic image annotation I.
Multigraph%2DBased Query%2DIndependent Learning for Video Search.

VideoSense A Contextual In%2DVideo Advertising System.

Two%2DDimensional Multilabel Active Learning with an Efficient Online Adaptation Model for Image Classification.

Graph%2Dbased semi%2Dsupervised learning with multiple labels.

Combining global
More and more computers are being incorporated in devices where the available amount of memory is limited. As a result research is increasingly focusing on the automated reduction of program size. Existing literature focuses on either data or code compaction or on highly language dependent techniques. The main contribution of this paper is to show how combined code and data compaction can be achieved using a link-time code compaction system that works by reasoning about the use of both code and data addresses. The analyses proposed are built on fundamental properties of linked code and therefore generally applicable. The combined code and data compaction is evaluated on SPEC2000, MediaBench and 4 C++ programs, resulting in binary program size reductions of 27%{53%. This compaction involves no speed trade-o, as the compacted programs are on average about 3% faster.  1. 
Video semantic analysis based on structure%2Dsensitive anisotropic manifold ranking.

Video semantic analysis based on structure%2Dsensitive anisotropic manifold ranking.

Multi%2Dvideo synopsis for video representation.

Video collage presenting a video sequence using a single image.

ImageSense.
We describe an unusual data set of thousands of annotated images with interesting sense phenomena. Natural language image sense annotation involves increased semantic complexities compared to disambiguating word senses when annotating text. These issues are discussed and illustrated, including the distinction between word senses and iconographic senses. 1
Bayesian video search reranking.

Online multi%2Dlabel active annotation towards large%2Dscale content%2Dbased video search.

Flickr distance.
Abstract. In this paper, we describe the participation of TELECOM ParisTech in the ImageCLEF 2010 Photo Annotation challenge. This edition focuses on promoting combination between visual and tag features in order to enhance photo annotation. An image collection is supplied with tags which are used both for training and testing. Our training approach consists of building SVM classifiers and kernels which take into account the similarity between visual features as well as tags. The results clearly corroborate (i) the complementarity of tags and visual descriptors and (ii) the effectiveness of SVM classifiers in photo annotation. 1
Contextual in%2Dimage advertising.

Annotating personal albums via web mining.

Finding image exemplars using fast sparse affinity propagation.

Study on the combination of video concept detectors.

A joint appearance%2Dspatial distance for kernel%2Dbased image categorization.

Coherent image annotation by learning semantic distance.

Coherent image annotation by learning semantic distance.

Joint multi%2Dlabel multi%2Dinstance learning for image classification.

Normalized tree partitioning for image segmentation.

Two%2DDimensional Active Learning for image classification.

Maximum Margin Clustering with Pairwise Constraints.

Video%5EM Multi%2Dvideo Synopsis.

Augmented tree partitioning for interactive image segmentation.

Query%2Dindependent learning for video search.

Graph%2Dbased semi%2Dsupervised learning with multi%2Dlabel.

Unbiased active learning for image retrieval.

Transductive video annotation via local learnable kernel classifier.

Smart video player.

Learning to video search rerank via pseudo preference feedback.

Optimized video scene segmentation.

Automatic video annotation through search and mining.
Conventional approaches to video annotation predominantly focus on supervised identification of a limited set of concepts, while unsupervised annotation with infinite vocabulary remains unexplored. This work aims to exploit the overlap in content of news video to automatically annotate by mining similar videos that reinforce, filter, and improve the original annotations. The algorithm employs a two-step process of search followed by mining. Given a query video consisting of visual content and speech-recognized transcripts, similar videos are first ranked in a multimodal search. Then, the transcripts associated with these similar videos are mined to extract keywords for the query. We conducted extensive experiments over the TRECVID 2005 corpus and showed the superiority of the proposed approach to using only the mining process on the original video for annotation. This work represents the first attempt at unsupervised automatic video annotation leveraging overlapping video content. Index Terms — video annotation, video search, data mining 1.
Free%2DShaped Video Collage.
tsinghua.edu.cn Abstract. With the explosive growth of multimedia data, video presentation has become an important technology for fast browsing of video content. In this paper, we present a novel video presentation technique called “Free-Shaped Video Collage ” (FS-Collage), which is motivated from and built upon our previous work on Video Collage [3]. Video Collage is a kind of static summary which selects the most representative regions-of-interest (ROI) from video and seamlessly arranges them on a synthesized image. Unlike Video Collage in which both the shapes of ROI and final collage are fixed as rectangle, we support arbitrary shapes of ROI and a set of collage templates in FS-Collage. Furthermore, we design three ROI arrangement schemes (i.e., book, diagonal, and spiral) for satisfying different video genres. We formulate the generation of FS-Collage as an energy minimization problem and solve the problem by designing a random sampling process. The experiment results show that our FS-Collage achieves satisfying performance. Key words: video collage, region of interest. 1
When multimedia advertising meets the new Internet era.

MILC2 A Multi%2DLayer Multi%2DInstance Learning Approach to Video Concept Detection.
Abstract. Video is a kind of structured data with multi-layer (ML) information, e.g., a shot is consisted of three layers including shot, keyframe, and region. Moreover, multi-instance (MI) relation is embedded along the consecutive layers. Both the ML structure and MI relation are essential for video concept detection. The previous work [5] dealt with ML structure and MI relation by constructing a MLMI kernel in which each layer is assumed to have equal contribution. However, such equal weighting technique cannot well model MI relation or handle ambiguity propagation problem, i.e., the propagation of uncertainty of sub-layer label through multiple layers, as it has been proved that different layers have different contributions to the kernel. In this paper, we propose a novel algorithm named MILC 2 (Multi-Layer Multi-Instance Learning with Inter-layer Consistency Constraint.) to tackle the ambiguity propagation problem, in which an inter-layer consistency constraint is explicitly introduced to measure the disagreement of inter-layers, and thus the MI relation is better modeled. This learning task is formulated in a regularization framework with three components including hyper-bag prediction error, inter-layer inconsistency measure, and classifier complexity. We apply the proposed MILC 2 to video concept detection over TRECVID 2005 development corpus, and report better performance than both standard Support Vector Machine based and MLMI kernel methods. 1
When multimedia advertising meets the new Internet era.

Optimizing video search reranking via minimum incremental information loss.

Transductive multi%2Dlabel learning for video concept detection.

Collaborative learning for image and video annotation.

MSRA atT TRECVID 2008 High%2DLevel Feature Extraction and Automatic Search.

Content%2DBased Multimedia Retrieval.
Unlike traditional database management systems, in multimedia databases that support content based retrieval over multimedia objects, it is difficult for users to express their exact information need directly in the form of a precise query. A typical interface supported by content-based retrieval systems allows users to express their information need in the form of examples of objects similar to the ones they wish to retrieve. Such a user interface, however, requires mechanisms to learn the query representation from the examples provided by the user. In this paper, we describe the generic query refinement framework we have implemented in the Multimedia Analysis and Retrieval System (MARS) for learning query representations using relevance feedback from users. The proposed framework uses a query expansion approach towards modifying the query representation in which relevant objects are added to the query. Furthermore, query reweighting techniques are used to adjust distance func...
Media Content Analysis.
Abstract—aceMedia content analysis capabilities are centered around the concept of the ACE. The ACE is composed of a content layer, a metadata layer and an intelligence layer. In this paper we show one application of the ACE Intelligence layer and how its proactive conduct can help in the complex task of adding semantic metadata to multimedia content. Index Terms—multimedia content analysis, proactive content, self-annotation I.
Correlative multilabel video annotation with temporal kernels.

VideoSense towards effective online video advertising.

Correlative multi%2Dlabel video annotation.
This paper describes the MSRA-USTC-SJTU experiments for TRECVID 2007. We performed the experiments in high-level feature extraction and automatic search tasks. For high-level feature extraction, we investigated the benefit of unlabeled data by semi-supervised learning, and the multi-layer (ML) multi-instance (MI) relation embedded in video by MLMI kernel, as well as the correlations between concepts by correlative multi-label learning. For automatic search, we fuse text, visual example, and concept-based models while using temporal consistency and face information for re-ranking and result refinement. Index Terms — support vector machines, semi-supervised learning, manifold ranking, multi-layer multi-instance kernel, linear neighborhood propagation, temporally consistent Gaussian random field, optimal multi-graph learning, correlative multi-label annotation, video annotation, video search. 1.
Video search re%2Dranking via multi%2Dgraph propagation.
This paper 1 is concerned with the problem of multimodal fusion in video search. First, we employ an object-sensitive approach to query analysis to improve the baseline result of text-based video search. Then, we propose a PageRank-like graph-based approach to text-based search result re-ranking. To better exploit the underlying relationship between video shots, the proposed reranking scheme simultaneously leverages textual relevancy, semantic concept relevancy, and low-level-feature-based visual similarity. In this PageRank-like scheme, we construct a set of graphs with the video shots as vertexes, and the conceptual and visual similarity between video shots as “hyperlinks. ” A modified topic-sensitive PageRank algorithm is then applied on these graphs to propagate the relevance scores through all related video shots. Experimental results verify the effectiveness of the graphbased propagation approach combined with the object-sensitive query analysis approach, which brings significant improvement to the baseline of text-based video search. Our experimental analysis also indicates that the proposed re-ranking method is highly generic and independent of different query classes, training data, and human interference.
Refining video annotation by exploiting pairwise concurrent relation.
In order to achieve true content-based information retrieval on video we should analyse and index video with high-level semantic concepts in addition to using user-generated tags and structured metadata like title, date, etc. However the range of such high-level semantic concepts, detected either manually or automatically, is usually limited compared to the richness of information content in video and the potential vocabulary of available concepts for indexing. Even though there is work to improve the performance of individual concept classifiers, we should strive to make the best use of whatever partial sets of semantic concept occurrences are available to us. We describe in this paper our method for using association rule mining to automatically enrich the representation of video content through a set of semantic concepts based on concept co-occurrence patterns. We describe our experiments on the TRECVid 2005 video corpus annotated with the 449 concepts of the LSCOM ontology. The evaluation of our results shows the usefulness of our approach.
Typicality ranking via semi%2Dsupervised multiple%2Dinstance learning.

Refining video annotation by exploiting pairwise concurrent relation.
In order to achieve true content-based information retrieval on video we should analyse and index video with high-level semantic concepts in addition to using user-generated tags and structured metadata like title, date, etc. However the range of such high-level semantic concepts, detected either manually or automatically, is usually limited compared to the richness of information content in video and the potential vocabulary of available concepts for indexing. Even though there is work to improve the performance of individual concept classifiers, we should strive to make the best use of whatever partial sets of semantic concept occurrences are available to us. We describe in this paper our method for using association rule mining to automatically enrich the representation of video content through a set of semantic concepts based on concept co-occurrence patterns. We describe our experiments on the TRECVid 2005 video corpus annotated with the 449 concepts of the LSCOM ontology. The evaluation of our results shows the usefulness of our approach.
Multi%2Dlayer multi%2Dinstance kernel for video concept detection.

Video collage.
tsinghua.edu.cn Abstract. With the explosive growth of multimedia data, video presentation has become an important technology for fast browsing of video content. In this paper, we present a novel video presentation technique called “Free-Shaped Video Collage ” (FS-Collage), which is motivated from and built upon our previous work on Video Collage [3]. Video Collage is a kind of static summary which selects the most representative regions-of-interest (ROI) from video and seamlessly arranges them on a synthesized image. Unlike Video Collage in which both the shapes of ROI and final collage are fixed as rectangle, we support arbitrary shapes of ROI and a set of collage templates in FS-Collage. Furthermore, we design three ROI arrangement schemes (i.e., book, diagonal, and spiral) for satisfying different video genres. We formulate the generation of FS-Collage as an energy minimization problem and solve the problem by designing a random sampling process. The experiment results show that our FS-Collage achieves satisfying performance. Key words: video collage, region of interest. 1
VideoSense a contextual video advertising system.

Structure%2Dsensitive manifold ranking for video concept detection.

Optimizing multi%2Dgraph learning towards a unified video annotation scheme.

Online video recommendation based on multimodal fusion and relevance feedback.
With Internet delivery of video content surging to an unprecedented level, video recommendation has become a very popular online service. The capability of recommending relevant videos to targeted users can alleviate users ’ efforts on finding the most relevant content according to their current viewings or preferences. This paper presents a novel online video recommendation system based on multimodal fusion and relevance feedback. Given an online video document, which usually consists of video content and related information (such as query, title, tags, and surroundings), video recommendation is formulated as finding a list of the most relevant videos in terms of multimodal relevance. We express the multimodal relevance between two video documents as the combination of textual, visual, and aural relevance. Furthermore, since different video documents have different weights of the relevance for three modalities, we adopt relevance feedback to automatically adjust intra-weights within each modality and inter-weights among different modalities by users ’ click-though data, as well as attention fusion function to fuse multimodal relevance together. Unlike traditional recommenders in which a sufficient collection of users ’ profiles is assumed available, this proposed system is able to recommend videos without users’ profiles. We conducted an extensive experiment on 20 videos searched by top 10 representative queries from more than 13k online videos, reported the effectiveness of our video recommendation system.
Concurrent Multiple Instance Learning for Image Categorization.
We propose a new multiple instance learning (MIL) algorithm to learn image categories. Unlike existing MIL algorithms, in which the individual instances in a bag are assumed to be independent with each other, we develop concurrent tensors to explicitly model the inter-dependency between the instances to better capture image’s inherent semantics. Rank-1 tensor factorization is then applied to obtain the label of each instance. Furthermore, we formulate the classification problem in the Reproducing Kernel Hilbert Space (RKHS) to extend instance label prediction to the whole feature space. Finally, a regularizer is introduced, which avoids overfitting and significantly improves learning machine’s generalization capability, similar to that in SVMs. We report superior categorization performances compared with key existing approaches on both the COREL and the Caltech datasets. 1.
An Interactive Video Annotation Frameowrk with Multiple Modalities.

Temporally Consistent Gaussian Random Field for Video Semantic Analysis.

On Real%2DTime Detecting Duplicate Web Videos.

Temporally Consistent Gaussian Random Field for Video Semantic Analysis.

Video Collage A Novel Presentation of Video Sequence.
This paper presents an automatic procedure for constructing a compact synthesized collage from a video sequence. The synthesized image called Video Collage, is a kind of static video summary – to select most representative images from video, to extract salient regions of interest (ROI) from these images and resize them according to their saliency, and to seamlessly arrange ROI on a given canvas with the temporal structure of video content preserved. We formulate the generation of Video Collage as an energy minimization problem in which each of above desirability is represented by an energy term. Unlike most existing video presentation schemes, Video Collage is more compact and visually appealing. We have applied Video Collage on several home videos and report superior performance in a user study compared with key existing approaches to video presentation. 1.
A Novel Multiple Instance Learning Approach for Image Retrieval Based on Adaboost Feature Selection.

Multi%2DGraph Semi%2DSupervised Learning for Video Semantic Feature Extraction.

Anisotropic Manifold Ranking for Video Annotation.

EMS Energy Minimization Based Video Scene Segmentation.

Lazy Learning Based Efficient Video Annotation.

Transductive Inference with Hierarchical Clustering for Video Annotation.

Beyond Accuracy Typicality Ranking for Video Annotation.

RMulti%2DConcept Multi%2DModality Active Learning for Interactive Video Annotation.

Building a comprehensive ontology to refine video concept detection.
Recent research has discovered that leveraging ontology is an effective way to facilitate semantic video concept detection. As an explicit knowledge representation, a formal ontology definition usually consists of a lexicon, properties, and relations. In this paper, we present a comprehensive representation scheme for video semantic ontology in which all the three components are well studied. Specifically, we leverage LSCOM to construct the concept lexicon, describe concept property as the weights of different modalities which are obtained manually or by data-driven approach, and model two types of concept relations (i.e., pairwise concept correlation and hierarchical relation). In contrast with most existing ontologies which are only focused on one or two components for domain-specific videos, the proposed ontology is more comprehensive and general. To validate the effectiveness of this ontology, we further apply it to video concept detection. The experiments on TRECVID 2005 corpus have demonstrated a superior performance compared to existing key approaches to video concept detection.
An Efficient Automatic Video Shot Size Annotation Scheme.

Video Histogram A Novel Video Signature for Efficient Web Video Duplicate Detection.
Abstract. The explosive growth of information technology and digital content industry stimulates various video applications over the Internet. Since it is quite easy to copy, reformat, modify and republish video files on the websites, similarity/duplicate detection and measurement is essential to identify the excessive content duplication, so as to facilitate effective video search and intelligence propriety protection as well. In this paper, we propose a novel signature-based approach for duplicate video comparison. The so-called video histogram scheme counts the numbers of video’s frames that are closest to a set of representative seed vectors chosen from the feature space of the training data set in advance. Then all the numbers are normalized to generate the signature of the video for further comparison. As our signature is a compact fixed-size vector with low dimension for each video, it requires less storage and computation cost than previous methods. The experiments show that our approach is both efficient and effective for web video duplicate detection. 1
Building a comprehensive ontology to refine video concept detection.
Recent research has discovered that leveraging ontology is an effective way to facilitate semantic video concept detection. As an explicit knowledge representation, a formal ontology definition usually consists of a lexicon, properties, and relations. In this paper, we present a comprehensive representation scheme for video semantic ontology in which all the three components are well studied. Specifically, we leverage LSCOM to construct the concept lexicon, describe concept property as the weights of different modalities which are obtained manually or by data-driven approach, and model two types of concept relations (i.e., pairwise concept correlation and hierarchical relation). In contrast with most existing ontologies which are only focused on one or two components for domain-specific videos, the proposed ontology is more comprehensive and general. To validate the effectiveness of this ontology, we further apply it to video concept detection. The experiments on TRECVID 2005 corpus have demonstrated a superior performance compared to existing key approaches to video concept detection.
Multi%2Dmodality web video categorization.

Kernel%2DBased Linear Neighborhood Propagation for Semantic Video Annotation.

VideoReach an online video recommendation system.
This paper presents a novel online video recommendation system called VideoReach, which alleviates users ’ efforts on finding the most relevant videos according to current viewings without a sufficient collection of user profiles as required in traditional recommenders. In this system, video recommendation is formulated as finding a list of relevant videos in terms of multimodal relevance (i.e. textual, visual, and aural relevance) and user click-through. Since different videos have different intra-weights of relevance within an individual modality and inter-weights among different modalities, we adopt relevance feedback to automatically find optimal weights by user click-though, as well as an attention fusion function to fuse multimodal relevance. We use 20 clips as the representative test videos, which are searched by top 10 queries from more than 13k online videos, and report superior performance compared with an existing video site.
MSRA%2DUSTC%2DSJTU AT TRECVID 2007 HIGH%2DLEVEL FEATURE EXTRACTION AND SEARCH.
This paper describes the MSRA-USTC-SJTU experiments for TRECVID 2007. We performed the experiments in high-level feature extraction and automatic search tasks. For high-level feature extraction, we investigated the benefit of unlabeled data by semi-supervised learning, and the multi-layer (ML) multi-instance (MI) relation embedded in video by MLMI kernel, as well as the correlations between concepts by correlative multi-label learning. For automatic search, we fuse text, visual example, and concept-based models while using temporal consistency and face information for re-ranking and result refinement. Index Terms — support vector machines, semi-supervised learning, manifold ranking, multi-layer multi-instance kernel, linear neighborhood propagation, temporally consistent Gaussian random field, optimal multi-graph learning, correlative multi-label annotation, video annotation, video search. 1.
Home Video Visual Quality Assessment With Spatiotemporal Factors.

Modeling and Mining of Users%27 Capture Intention for Home Videos.
Abstract — With rapid adoption of consumer digital video recorders and increase of home video data, content analysis has become an interesting and key research issue to provide personalized experiences and services for both camcorder users and viewers. In this paper, we present a novel view to tackle this issue, which aims at modeling and mining of the capture intention of camcorder users. Based on the study of intention mechanism in psychology, a set of domain-specific capture intention concepts is defined. A comprehensive and extensible scheme consisting of video structure decomposition, intention oriented feature analysis, as well as SVD-based intention segmentation and learning-based intention classification is proposed to mine the users ’ capture intention. Experiments were carried on home video sequences of 90 hours in total, taken by 16 persons over the past 20 years. Both the user study and objective evaluations indicate that our proposed intention-based approach is an effective complement to existing home video content analysis schemes. Index Terms — Video content analysis, capture intention, attention detection. I.
Interactive Video Annotation by Multi%2DConcept Multi%2DModality Active Learning.

Manifold%2Dranking based video concept detection on large database and feature pool.

Towards content%2Dbased relevance ranking for video search.

To construct optimal training set for video annotation.

Automatic video annotation by semi%2Dsupervised learning with kernel density estimation.

Semi%2DSupervised Kernel Regression.
The paper presents a semi-supervised kernel method for regression estimation in the presence of unlabeled patterns. The method exploits a recently proposed datadependent kernel which is constructed in order to represent the inner geometry of the data. This kernel is implemented into Kernel Regression methods (SVR, KRR). Experimental results aim to highlight the properties of the method and its advantages as compared to fully supervised approaches. The influence of the parameters on the model properties was evaluated experimentally. One artificial and two real-world datasets were used to demonstrate the performance of the proposed algorithm. 1.

Automatic Video Genre Categorization using Hierarchical SVM.
This paper presents an automatic video genre categorization scheme based on the hierarchical ontology on video genres. Ten computable spatio-temporal features are extracted to distinguish the different genres using a hierarchical Support Vector Machines (SVM) classifier built by cross-validation, which consists of a series of SVM classifiers united in a binary-tree form. As the order and genre partition strategy of the SVM classifier series affect the over performance of the united classifier, two optimal SVM binary trees, local and global, are constructed aiming at finding the best categorization orders, i.e., the best tree structure, of the genre ontology. Experimental results show that the proposed scheme outperforms C4.5 Decision Tree, typical 1-vs-1 SVM scheme, as well as the hierarchical SVM built by K-means. Index Terms — Pattern Classification, Video Signal Processing 1.

Probabilistic Multimodality Fusion for Event based Home Photo Clustering.

Video Annotation by Active Learning and Semi%2DSupervised Ensembling.

Automatic video annotation based on co%2Dadaptation and label correction.

A semi%2Dsupervised incremental learning framework for sports video view classification.

Efficient semantic annotation method for indexing large personal video database.

Intention%2Dbased home video browsing.
This demonstration presents an e#cient home video browsing system from a novel viewpoint -- capture intention. We extend our previous work to build up a comprehensive scheme  to mine the capture intention, and based on this scheme, propose a novel home video browsing system. Such an intention based system assists both camcorder users and viewers to experience their personal home videos by providing two better manners of browsing -- by intention thumbnails and by intention curves. The user study indicates that our system o#ers users a more e#cient way to search a given clip in a video than attention-based scheme.
Natural video browsing.
In this demonstration, we show a novel system, Video Booklet, which enables nature personal video browsing and searching.
Personal media sharing and authoring on the web.

To learn representativeness of video frames.
With the rapid explosion of video data, compact representation of videos is becoming more and more desirable for e#- cient browsing and communication, which leads to a number of research works on video summarization in recent years. Among these works, summaries based on a set of still frames are frequently studied and applied due to its high compactness. However, the representativeness of the selected frames, which are taken as the compact representation of the video or video segment, has not been well studied. It is observed that frame representativeness is highly related to the following elements: image quality, user attention measure, visual details, and displaying duration. It is also observed that users have similar tendency in selecting the most representative frame for a certain video segment. In this paper, we developed a method to examine and evaluate the representativeness of video frames based on learning users&#039; perceptive evaluations.
Spatio%2Dtemporal quality assessment for home videos.
Compared with the video programs taken by professionals, home videos are always with low-quality content resulted from lack of professional capture skills. In this paper, we present a novel spatio-temporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, sub-shot, is selected as the basic unit for quality assessment. A set of spatio-temporal artifacts, regarded as the key factors a#ecting the overall perceived quality (i.e. unstableness, jerkiness, infidelity, blurring, brightness and orientation), are mined from each sub-shot based on the particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three di#erent methods, including user study, factor fusion, and a learningbased scheme. To validate the proposed scheme, we present a scalable quality-based home video summarization system, aiming at achieving the best quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model based video skimming approach demonstrated the e#ectiveness of the proposed quality assessment scheme.
Tracking users%27 capture intention a novel complementary view for home video content analysis.
In this paper, we present a novel view to home video content analysis, which aims at tracking the capture intention of camcorder users. Based on the study of intention mechanism in psychology, a set of domain-specific capture intention concepts are defined. A comprehensive and extensible scheme consisting of video structuring, intention oriented feature analysis, as well as intention unit segmentation and classification is proposed to mine the users ’ capture intention. Experiments were carried on home video sequences of 90 hours in total, taken by 16 persons in recent 20 years. Both the user study and objective evaluations indicate that our proposed intention-based approach is an effective complement to existing home video content analysis schemes.
LazyCut content%2Daware template%2Dbased video authoring.

Efficient video mosaicing based on motion analysis.
Presenting more comprehensive information than key frame and any subset of frames, mosaic has attracted a growing attention in recent years as a useful element for a variety of vision tasks. In this paper, we present an efficient approach to video mosaicing based on motion analysis with investigating two important issues that significantly affect the performance of mosaicing. First, given a video shot, a method based on motion entropy analysis is introduced to determine whether the visual content in a shot or video segment is suited to be represented by a mosaic. The second is to decide which subset of frames in the shot is the best to be selected for efficient and effective mosaicing by constructing a global motion path. Experimental comparisons with typical mosaicing method approve that with this approach, the visual quality of mosaic is significantly improved, as well as the computation time is remarkably reduced.
Online End Detection for Live%2DBroadcast Sports TV Programs.

Robust learning%2Dbased TV commercial detection.

Video booklet.
In this demonstration, we show a novel system, Video Booklet, which enables nature personal video browsing and searching.
Camera notes.

Video booklet a natural video searching and browsing interface.

Tracking concept drifting with an online%2Doptimized incremental learning framework.

Semi%2Dautomatic video annotation based on active learning with multiple complementary predictors.
In this paper, we will propose a novel semi-automatic annotation scheme for video semantic classification. It is well known that the large gap between high-level semantics and low-level features is difficult to be bridged by full-automatic content analysis mechanisms. To narrow down this gap, relevance feedback has been introduced in a number of literatures, especially in those works addressing the problem of image retrieval. And at the same time, active learning is also suggested to accelerate the converging speed of the learning process by labeling the most informative samples. Generally an active learning scheme includes a sample selection engine and a learning engine. In this paper, we will discuss the limitations of existing active learning algorithms and propose a novel active learning scheme based on multiple complementary predictors and incremental model adaptation, which improves the efficiencies of both of the primary components of active learning. Firstly, an efficient sample selection scheme is proposed, in which multiple predictors are applied to find most informative samples. Then an incremental model adaptation technique, maximum likelihood linear regression (MLLR), is used to update the classifiers which tackle the issue of unbalance between the original training set and the newly labeled data. It is proved that the samples selected by the proposed scheme are more representative than general active learning scheme, as well as the incremental model adaptation scheme is effective especially when the newly added data size is small.
A generic framework of user attention model and its application in video summarization.

P%2DKaraoke personalized karaoke system.
In this demonstration, a personalized Karaoke system, P-Karaoke, is proposed. In the P-Karaoke system, personal home videos and photographs, which are automatically selected from users’ multimedia database according to their content, users ’ preferences or music, are utilized as the background videos of the Karaoke. The selected video clips, photographs, music and lyrics are well aligned to compose a Karaoke video, connecting by specific content-based transitions.
An online%2Doptimized incremental learning framework for video semantic classification.
This paper considers the problems of feature variation and concept uncertainty in typical learning-based video semantic classification schemes. We proposed a new online semantic classification framework, termed OOIL (for Online-Optimized Incremental Learning), in which two sets of optimized classification models, local and global, are online trained by sufficiently exploiting both local and global statistic characteristics of videos. The global models are pre-trained on a relatively small set of pre-labeled samples. And the local models are optimized for the under-test video or video segment by checking a small portion of unlabeled samples in this video, while they are also applied to incrementally update the global models. Experiments have illustrated promising results on simulated data as well as real sports videos.
Automatic music video generation based on temporal pattern analysis.
Music video (MV) is a short film meant to present a visual representation of a popular music song. In this paper, we present a system that automatically generates MV-like videos from personal home videos based on observations that generally there are obvious repetitive visual and aural patterns in MVs. Based on a set of video and music analysis algorithms, the automatic music video (AMV) generation system automatically extracts temporal structures of the video and music, as well as repetitive patterns in the music. And then, according to the structure and patterns, a set of highlight segments from the raw home video footage are selected, aiming at matching the visual content with the aural structure and pattern. And last, the output music video is rendered by connecting the selected highlight video segments with appropriate transition effects, accompanied with the music. Experiments show that the results are compelling and promising.
Automatically converting otograic series into video.

Content and transformation effect matching for automated home video editing.

Robust video signature based on ordinal measure.
In this paper we proposed a video signature based on ordinal measure of resampled video frames, which is robust to changing compression formats, compression ratios, frame sizes and frame rates. To effectively localize a short query video clip in a long target video through the proposed video signature, we developed a coarse-to-fine signature comparison scheme. In the coarse searching step, roughly matched positions are determined based on Sequence Shape Similarity, while in the fine searching step, dynamic programming is applied to handle similarity matching in the cases of losing frames and temporal editing processes are employed on the target video. Experiments showed that the proposed video signature has good robustness and uniqueness, which are the two essential properties of video signatures. 1.
Online Play Segmentation for Broadcasted American Football TV Programs.
This paper proposes an online segmentation scheme for broadcasted  American football TV programs. The TV video streams are online  partitioned into a series of play-break patterns with a relatively short  delay. All the detected plays form an event-based summarization which  contains complete information about the original match. The main contributions  of the proposed scheme are twofold. First, an online adaptive  field-color model is employed to deal with feature variations which may  caused by court configurations and time-varying lighting conditions. Second,  a temporal pattern based model is constructed to capture intrinsic  features of football plays, which considerably increased the performance  of the proposed scheme. Effectiveness of our approach is demonstrated  by experiments on real recorded TV programs.
An Attention%2DBased Decision Fusion Scheme for Multimedia Information Retrieval.
In this paper, we proposed a novel decision fusion scheme  based on the psychological observations on human beings&#039; visual and  aural attention characteristics, which combines a set of decisions obtained  from di#erent data sources or features to generate better decision result.
An Online Learning Framework for Sports Video View Classification.

An automatic performance evaluation protocol for video text detection algorithms.
Text presented in videos provides important supplemental information for video indexing and retrieval. Many efforts have been made for text detection in videos. However, there is still a lack of performance evaluation protocols for video text detection. In this paper, we propose an objective and comprehensive performance evaluation protocol for video text detection algorithms. The protocol includes a positive set and a negative set of indices at the textbox level, which evaluate the detection quality in terms of both location accuracy and fragmentation of the detected textboxes. In the protocol, we assign a detection difficulty (DD) level to each ground truth textbox. The performance indices can then be normalized with respect to the textbox DD level and are therefore tolerant to different ground-truth difficulties to a certain degree. We also assign a detectability index (DI) value to each ground-truth textbox. The overall detection rate is the DI-weighted average of the detection qualities of all ground-truth textboxes, which makes the detection rate more accurate to reveal the real performance. The automatic performance evaluation scheme has been applied to performance evaluation of a text detection approach to determine the best thresholds that can yield the best detection results. The protocol has also been employed to compare the performances of several text detection systems. Hence, we believe that the proposed protocol can be used to compare the performance of different video/image text detection algorithms/systems and can even help improve, select, and design new text detection methods.
Optimization%2Dbased automated home video editing system.
Abstract—In this paper, we present an optimization-based system that automates home video editing (AVE). This system automatically selects suitable or desirable highlight segments from a set of raw home videos and aligns them with a given piece of incidental music to create an edited video segment to a desired length based on the content of the video and incidental music. We developed an approach for extracting temporal structure and determining the importance of a video segment in order to facilitate the selection of highlight segments. Additionally we extract temporal structure, beats and tempos from the incidental music. In order to create more professional-looking results, the selected highlight segments satisfy a set of editing rules and are matched to the content of the incidental music. This task is formulated as a non-linear 0-1 programming problem and the rules, which are adjustable and increasable, are embedded as constraints. The output video is rendered by connecting the selected highlight video segments with transition effects and the incidental music. Under this framework, we can choose the best-matched music for a given video and support different output styles.
AVE automated home video editing.
In this paper, we present a system that automates home video editing. This system automatically extracts a set of highlight segments from a set of raw home videos and aligns them with user supplied incidental music based on the content of the video and incidental music. We developed an approach for extracting temporal structure and determining the importance of a video segment in order to facilitate the selection of highlight segments. Additionally we extract temporal structure, beats and tempos from the incidental music. In order to create more professional-looking results, the selected highlight segments satisfy a set of editing rules and are matched to the content of the incidental music. This task is formulated as a non-linear 0-1 programming problem and the rules are embedded as constraints. The output video is rendered by connecting the selected highlight video segments with transition effects and the incidental music.
Photo2Video.
To exploit rich content embedded in a single photograph, a system named Photo2Video was developed to automatically convert a photographic series into a video by simulating camera motions, set to incidental music of the user&#039;s choice. For a chosen photographic series, an appropriate camera motion pattern is selected for each photograph to generate a corresponding motion photograph clip. Then, the final output video is rendered by connecting a series of motion photograph clips with specific transitions, and aligning with the selected incidental music. Photo2Video provides a novel way to browse a series of images and can be regarded as a system exploring the new medium between photograph and video.
Content based photograph slide show with incidental music.
In this paper, we proposed a new scheme to generate photograph slide show with incidental music based on the content of the photographs and music. Unlike existing photograph slide show software, which show the photographs one by one (perhaps with some fancy transition effects) with music in the background, our algorithm utilizes content analysis of the photographs and the incidental music. Firstly, low-quality photographs, such as blurred or under exposed, are filtered out by a photograph quality detector; and the rest of the photographs are grouped into a three-layer construction, namely, day, scene, and group of similar photographs (GoS). Incidental music is segmented based on its beats, and then aligned with photographs, which makes the photograph transitions occur at the beat position of the music. Various transition effects and transformation effects are applied based on the structure and content of the photographs. According to different users&#039; preferences, several editing styles are also supported in our algorithm. The experimental results are very impressive according to the user study.
Efficient video text recognition using multiple frame integration.
Text superimposed on the video frames provides supplemental but important information for video indexing and retrieval. Many efforts have been made for videotext detection and recognition (Video OCR). The main difficulties of video OCR are the low resolution and the background complexity. In this paper, we present efficient schemes to deal with the second difficulty by sufficiently utilizing multiple frames that contain the same text to get every clear word from these frames. Firstly, we use multiple frame verification to reduce text detection false alarms. And then choose those frames where the text is most likely clear, thus it is more possible to be correctly recognized. We then detect and joint every clear text block from those frames to form a clearer &#034;man-made&#034; frame. Later we apply a block-based adaptive thresholding procedure on these &#034;man-made&#034; frames. Finally, the binarized frames are sent to OCR engine for recognition. Experiments show that the word recognition rate has been increased over 28% by these methods.
Automatic Performance Evaluation for Video Text Detection.
In this paper, we propose an objective, comprehensive and difficulty-independent performance evaluation protocol for video text detection algorithms. The protocol includes a positive set and a negative set of indices at textbox level, which evaluate the detection quality in terms of both location accuracy and fragmentation of the detected textboxes. In the protocol, we assign a detection difficulty (DD) level to each ground truth textbox. The performance indices can then be normalized with respect to the textbox DD level and are therefore independent of the ground truth difficulty. We also assign a detection importance (DI) level to each ground truth textbox. The overall detection rate is the DI-weighted average of the detection qualities of all ground truth textboxes, which makes the detection rate more accurate to reveal the real performance. The automatic performance evaluation scheme has been applied on a text detection approach to determine the best parameters that can yield the best detection results.
A Video Text Detection And Recognition System.
In this paper, we propose a new system for text information extraction from news videos. First of all, a method that integrates text detecting and text tracking is developed to locate text areas in the key-frames (images), together with a scheme to evaluate the performance of this approach. To get better recognition results, we then enhance the quality of the detected text blocks by multi-frame averaging. Finally, we use an adaptive thresholding method to binarize the text blocks and recognize the text using an off-the-shelf OCR module. The detection and recognition rate of the proposed system are 94.7 % and 67.5 % respectively. 1.
